Image processing

ABSTRACT

A method of processing image data representing a captured image, the data generated using an image capture element of an image capture device, the method comprising generating state data representing at least one characteristic of motion of the image capture device and a status of at least one setting of the device during a predetermined time period, and processing the image data on the basis of the state data in order to provide a measure of the accuracy of framing of the captured image within a field of view of the device, and corresponding apparatus.

FIELD OF THE INVENTION

The present invention relates to the field of image processing.

BACKGROUND

When taking pictures a user of an image capture device such as a camerafor example may or may not put care into framing a subject of the imageaccurately. Hence, in some situations pictures are taken quickly and aregenerally referred to as ‘snaps’ or ‘snapshots’, whilst in others a longtime can be spent carefully framing a subject in order to obtain adesired image composition.

Differences in picture taking behaviour can be the result of acombination of factors. For example, some camera users always takepictures very quickly, perhaps due to lack of training or through notwishing to spend a long time “behind the camera”, whilst other userstake time and trouble when the situation permits or a good result iscritical, but on other occasions are forced to take a quick snap.

A further distinction in picture taking behaviour relates to the subjectitself. In the case of moving objects, a photographer may wait for adesired moment to capture an image. Some examples include: a subjectscoring a goal, eye-contact with the user, and the sun emerging frombehind a cloud and so forth. In these situations, attention is morefocused on capturing a desired moment in time, rather than ensuring thata subject of the captured image is well framed.

Current cameras do not recognize the picture taking behaviour of a user,other than by the direct operation of the camera controls. As such, theimage processing functionality of the camera is generally determinedonly by its settings and the content of the captured image. Thisprecludes the possibility of making image processing responsive to theway the camera is being used.

A particular problem with this rigid approach to image processing isthat processing steps such as autocropping are either ‘on’ or ‘off’, asgenerally set by user choice —the term “autocropping” refers toalgorithms which automatically generate one or more crops of aphotographic image based on an algorithmic assessment of likely areas ofinterest, or salient areas, within a captured image. The autocroplocation can be selected to include the areas of interest, removedistractions or large boring areas, and produce a pleasing compositionfor example.

A number of autocropping approaches are known. U.S. patent applicationSer. No. U.S. Pat. No. 5,978,519 is an example. It assumes that the“background” region of a captured image has relatively low intensityvariance compared with the foreground. A threshold is selected based onthe variance of blocks, and a bounding box of the subject is then usedfor cropping.

A more recent approach may be found in United Kingdom Patent ApplicationNo. GB0031423.7 which uses a saliency map determined by regionsegmentation and colour ‘unusual-ness’ to identify both subject areasand distraction areas. Multiple crops are suggested based on alternativecombinations of subjects, in each case excluding distractions andselecting locally optimal crop boundaries.

European Patent Application No. EP1158464A1 also uses a saliency mapderived from region segmentation.

Some of the above algorithms can have hints or constraints applied by anexternal source (typically a person interacting with the algorithm).These provide the algorithm with guidance to ensure that some areas arealways preserved, but do not provide an automatic determination of whenautocropping may be appropriate.

STATEMENT OF THE INVENTION

According to a first aspect of the present invention there is provided amethod of processing image data representing a captured image, the datagenerated using an image capture element of an image capture device, themethod comprising generating state data representing at least onecharacteristic of motion of the image capture device and a status of atleast one setting of the device during a predetermined time period, andprocessing the image data on the basis of the state data in order toprovide a measure of the accuracy of framing of the captured imagewithin a field of view of the device.

According to a second aspect of the present invention there is providedan image capture device comprising an image capture element operable togenerate image data, a processor, and a state data generator forgenerating state data representing at least one characteristic of motionof the image capture device and a status of at least one setting of thedevice during a predetermined time period, wherein the processor isoperable to process image data representing a captured image on thebasis of the state data in order to provide a measure of the accuracy offraming of the captured image within a field of view of the device.

According to a third aspect of the present invention there is providedan image capture device comprising an image capture element operable togenerate image data, a processor, and a state data generator forgenerating state data representing at least one characteristic of motionof the image capture device and a status of at least one setting of thedevice during a predetermined time period, the processor being operableto process image data representing a captured image on the basis of thestate data in order to provide crop data representing a measure of theaccuracy and/or type of framing of the captured image within a field ofview of the device, the processor operable to use the crop data in orderto determine a subset of the captured image data to delete from a memoryof the device.

According to a fourth aspect of the present invention there is provideda computer program product for use with a computer, said computerprogram product comprising a computer useable medium having computerexecutable program code embodied thereon, wherein said product isoperable, in association with said computer, to generate state datarepresenting at least one characteristic of motion of an image capturedevice and a status of at least one setting of the device during apredetermined time period, and process image data on the basis of thestate data in order to provide a measure of the accuracy of framing ofthe captured image within a field of view of the device.

According to a fifth aspect of the present invention there is provided acomputer program, comprising machine readable instructions, wherein saidprogram is arranged, in association with said machine, to generate statedata representing at least one characteristic of motion of an imagecapture device and a status of at least one setting of the device duringa predetermined time period, and process image data on the basis of thestate data in order to provide a measure of the accuracy of framing ofthe captured image within a field of view of the device.

BRIEF DESCRIPTION OF THE FIGURES

For a better understanding of the present invention, and to furtherhighlight the ways in which it may be brought into effect, variousembodiments will now be described, by way of example only, withreference to the following drawings in which:

FIG. 1 is a schematic representation of a camera operable in accordancewith an embodiment of the present method;

FIG. 2 is a schematic representation of exemplary stages of an imagecapture operation when using the camera of FIG. 1;

FIG. 3 is a flow diagram representing an exemplary procedure forprocessing data generated using the camera of FIG. 1 according to anembodiment; and

FIG. 4 is a schematic representation of an image captured using thecamera of FIG. 1.

It should be emphasized that the term “comprises/comprising” when usedin this specification specifies the presence of stated features,integers, steps or components but does not preclude the presence oraddition of one or more other features, integers, steps, components orgroups thereof.

DETAILED DESCRIPTION

FIG. 1 is a schematic representation of an image capture device 100suitable for use in an embodiment. Device 100 can be a digital still orvideo camera or a combination thereof, or any other suitable imagecapture device including a device comprising image capture functionalitysuch as a mobile telephone or PDA for example. Broken lines in FIG. 1indicate camera elements which are inside the camera body and thereforenot normally visible to a user. The camera comprises an image captureelement 101 such as a CCD or CMOS device, or any other suitable imagecapture element. The image capture element 101 is operable to generateimage data representing an image of a scene or object.

Sensor 103 of camera 100 is operable to generate data representing astate of the camera substantially prior to generation of the image data.Preferably, this state data generated by the sensor 103 relates to useof the camera and can therefore include data representing positionand/or orientation changes of the camera (camera movement, or movementpatterns for example) and/or the condition of camera settings forexample. According to a preferred embodiment, the state data isprocessed by a suitable processor 105 of the camera in order to providedata for use in interpreting the semantics of an image capturesituation. This will be explained in further detail below.

The time frame over which changes in cameraposition/orientation/settings occur can be important factor in theinterpretation. For example, when using generated state data tointerpret the semantics of an image capture procedure, more or lessweight may be given to certain actions and/or camera movements on thebasis of the time frame over which they have occurred. So for example aslow steady movement of the camera 100 can indicate that a user istaking care to frame image, whereas an abrupt movement of the camera canindicate that the a captured image will not be well framed due to thelack of time that has been spent framing a subject of the image.

The interpretation can also comprise an analysis of data representingthe motion of a subject in a scene of the image in question, and/or datarepresenting the use of camera functions, such as a zoom function of thecamera for example.

Inferring appropriate image constraints automatically from the behaviourof the camera user, and hence changes to a state or condition of thecamera, prior to image capture obviates the need for a user toexplicitly interact with an autocrop algorithm of the device. Inparticular, if processed state data indicates that a user has taken careto compose an image, an autocrop algorithm or similar image processingstep of the camera 100 can be disabled.

In a preferred embodiment, data representing the operation of a cameraduring a preview period of an image capture operation is analysed byprocessor 105 in order to determine a probability of a type of framingof a captured image and the accuracy of the framing. More specifically,movement of an image capture device and the operation of the devicecontrols, amongst other things, results in the generation of data whichis used in order to infer or classify a framing type and accuracythereof.

As mentioned above, a type of image framing—or more specifically, theway a subject or subjects of an image are framed within the field ofview of a camera—can be characterised by a use model of the camera 100.So, for example, if the camera is used to take a quick snap, the imageframing is likely to be less than optimal due to lack of care and/ortime in capturing an image such that the image is not framed in aportion of the field of view of the camera 100 which was intended forexample.

In this connection, sensor 103 can be any sensor suitable for generatingdata representing a position and/or orientation and/or setting of thecamera 100, and/or changes thereof. For example, sensor 103 may be anaccelerometer operable to generate data representing a movement of thecamera 100 in any one or more of the degrees of freedom of the camera.Alternatively, sensor 103 may be a suitably programmed processoroperable to determine the camera settings or a change therein andgenerate data representative thereof. Combinations of the above arepossible, as are other suitable alternatives as will be appreciated bythose skilled in the art. Camera 100 may include more than one sensor103, however for the sake of clarity the present description willreference the use of one suitable sensor in a device.

The time period before image capture in which state data is generated bythe sensor 103 can be characterised as a ‘preview period’, as it is thetime when a user of a camera frames an image before capture thereof.Such data can continue to be generated after an image has been capturedhowever, and can therefore also be used to provide any desiredpost-processing of image data.

In this connection, FIG. 2 is a schematic representation of exemplarystages of a typical hand-held image capture process in one dimension.More or less stages may be present in an image capture operation.

In stage 201 which is generally referred to as the aiming stage, acamera user attempts to get the aim of the camera 100 progressivelycloser to the target of which an image is desired using a combination ofballistic and visually servoed adjustments of the camera for example, inorder to place the desired subject of an image within a desired portionof the field of view of the camera 100.

In the second stage 203 the photographer attempts to steady the camera100 using both visual servoing and muscle feedback for example. Thethird stage 205 is the moment when the photographer presses the shutterbutton (or similar) of the camera 100 in order to effect image capture.The fourth stage 207 is the exposure time where an image is captured byan image capture element 101 of the camera 100. If manual zoom and focusfunctions are present on the camera 100, these may also be used duringthe above stages. The relative sizes of the stages are not to scale inthe figure.

The position of a notional subject in one dimension with respect to timeof which an image is desired is depicted by line 209 in FIG. 2, and is astationary target in the dimension being considered in the figure. Asmentioned above, motion of a desired subject can be taken into accountin order to help interpret the semantics of an image capture situation.For the sake of clarity, the present discussion will be limited to thecase that the subject of FIG. 2 is stationary in the dimension depictedin the figure. If the subject were moving in the dimension underconsideration, known subject/target tracking techniques could beemployed in order to determine a motion pattern of the subject and datarelating to this could be generated by a suitable sensor or processor ofthe camera, and used appropriately, such as, for example in compensatingfor motion of the subject when determining its desired position withinthe field of view of the camera, and its final position in a captureimage.

The line 211 represents the position or orientation of a camera 100 inone of the six degrees of freedom of the camera with respect to time.Line 211 can represent the position or orientation of an image captureelement 101 of a camera 100 with respect to time if it is the motion ororientation of this element of the camera which determines the framingtype and accuracy of a captured image.

In the example of FIG. 2, it is assumed that 211 represents motion ofthe camera 100 in a vertical plane. It will be appreciated that othermotions and/or orientations of the camera 100 (and hence the captureelement 101) are possible.

In the aiming stage 201, the camera 100 is moved in order to frame atarget. In the first stage of the aiming process of FIG. 2, which,broadly speaking, is covered by the period depicted by 213, a user movesthe camera upwards with respect to the target in order to frame thetarget, or adjusts the orientation accordingly. In the particularexample depicted by FIG. 2, the target will be substantially desirablyframed when the path 211 intercepts 209 as will be explained in furtherdetail below.

Generally in the period depicted by 215 of the aiming stage 201 the usermakes adjustments to the camera aim in order to keep the target in thedesired portion of the field of view of the camera 100, and/or achievemore desirable framing of the target.

In the steady period 203, the user attempts to keep the camerasubstantially aligned with the target in order to maintain the desiredframing for image capture. It will be noted that at this stage, thecamera motion line 211 substantially overlaps the target line 209 and isrelatively steady compared to the aiming stage thereby indicating that adesirable framing of the target of which an image is desired has beenestablished.

The deviation in the line 211 in the exposure stage 205 corresponds to auser pressing a button (not shown) on the camera 100 in order tofacilitate image capture. Such a button is generally on the top of thecamera, and so the line 211 is depicted as moving down in FIG. 2corresponding to movement of the camera 100 vertically down as thebutton is pressed. It will be appreciated that such a button may besituated anywhere on a camera 100, and the above is not intended to belimiting. Alternatively, image capture may be initiated by some othermeans such a voice activated command or some form of biometric signaturechange (pulse, perspiring, pupil dilation etc) in which case there maybe negligible deviation of the camera 100 during the period identifiedby 205. In general, stage 205 will be present unless the camera beingused is one which is continuously capturing a sequence of images such asa video camera for example, in which case, the stage 205 will not bepresent as indicated in FIG. 2. In that case, the division between thestages of FIG. 2 will be somewhat blurred as a user of a video camerawill generally be attempting to constantly maintain a desired accuracyof framing whilst capturing image data.

Stage 207 is the exposure stage in which image data representing thescene or object of interest is generated. In the example of FIG. 2, theposition of the camera 100 in the stage 207 (i.e. line 211) has deviatedfrom that of the steady stage 203 as a result of image capture actuationwhich as mentioned will generally be effected by depressing a suitablebutton or other such means. In general, the position/orientation of thecamera will deviate from that of the steady stage either due toactuation of a button for initiating image capture, or perhaps due to alack of concentration by a user following stage 205 which may be causedby the user thinking that the capture occurs substantiallysimultaneously with stage 205. In reality this is generally not thecase, and there will be a delay between image capture actuation andimage data generation which will vary from device to device.Furthermore, in certain situations, a longer exposure time will berequired (e.g. in low light) which can result in further camera movementduring stage 207.

Stages 201 and 203 taken together correspond broadly to a “preview”stage of an image capture operation mentioned above, in which a userdetermines and/or tracks a target and attempts to move/orient the camerain order to frame the target in a desirable way. Desirable framing candepend on the capture conditions and/or the nature of the target.

Data representing at least the patterns of motion and camera controlduring the preview stage of an image capture operation is generated bythe sensor 103, and such data is used in order to classify the framingof a target in a captured image and determine the accuracy of thisframing.

According to a preferred embodiment, three elements of an image captureprocess which results in an image can be addressed using the generatedstate data: i) framing care, ii) framing accuracy, and iii) framingtype.

Framing care can be characterised in terms of how much a user “thought”before framing a target, or how much time a camera aim was finelyadjusted before finally capturing the image.

Framing accuracy, which is generally only partly related to care, can becharacterised in terms of how likely it is that the framing of a targetscene or object that the user wanted was intentional or unintentional.Usually high “care” implies high “precision”, but the opposite is notalways true.

Framing type can be characterised in terms of whether there is likely tobe a subject and a foreground, whether the subject was centred orpositioned in the left or right of a captured image, whether it is faraway or close, etc. Many of these properties can be determined byanalyzing the data relating to the behaviour of the user and theiroperation of the camera.

So, for example, with reference to FIG. 2, data representing changes inthe position/orientation of the camera 100 in the preview period, andthe use of any functions of the camera by a user such as a zoom and/ormanual focus feature for example, is generated by sensor 103. Byanalysing this data, the portion/portions of a subsequently capturedimage which were intended to be framed by the user can be determined aswill explained below, and any necessary post capture processing can beapplied to the captured image data.

In order to determine the type of framing and the accuracy thereof, thestate data generated by the sensor 103 is processed by processor 105 ofcamera 100. The state data can represent any or all of position and/ororientation changes of the camera 100, the time taken to capture theimage 300 (or the time associated with any of the stages of FIG. 1), thetime spent focussed on a particular point in space, operation of thecamera zoom, and framing an object/scene and then moving away slightlyfor example. Further useful measures in addition to those listed aboveare possible as will be appreciated.

FIG. 3 of the accompanying drawings is a flow diagram representing anexemplary procedure for processing data according to an embodiment.

An image capture device 300 includes an image capture element 301 suchas one of the suitable elements mentioned above, and a sensor 303. Theelement 301 is operable to generate image data 305 representative of acaptured image. The sensor is operable to generate state data 307representative of a condition of device 300 substantially prior togeneration of data 305.

The state data 307, and optionally the image data 305 are processedusing a suitable processor 309. Processor 309 may be integral to device300 or remote, in which case the device comprises additionalfunctionality in order to effect transmission of data 305,307 asappropriate using known techniques such as wired or wireless datacommunications links for example.

Processor 309 is operable to send and receive data from a suitablememory of the device (not shown). Once data 305,307 as appropriate hasbeen processed, processor 309 generates output data 311. The data 311can be data representing a use model of the device 300, or can compriseoutput image data, in which case, processor 309 is operable to processdata 305 and data representing a use model of device 300 in order togenerate output data 311.

The use of state data 307 generated using sensor 103, 301 will now beexemplified with reference to FIG. 4 of the accompanying drawings whichis a schematic representation of a captured image 400. The image 400comprises a background, generally denoted by reference 401, and aforeground 403 which includes an object 405. The object 405 is thedesired subject of the image 400, whose motion in one dimension withrespect to time can be 209 of FIG. 2.

Motion of the camera 100 in stages 205,207 of FIG. 2 has resulted in thesubject 405 being less than optimally framed in a desired portion of thefield of view of the camera 100. In this case, the optimality of theframing may be determined from analysis of the steady stage 203. For thepurposes of the present example, it can be assumed that when line 211intercepts a target (209), the target is substantially centrally framedwithin the field of view of the camera in question, and that this is theframing which was desired by a user of the camera 100. Otheralternatives are possible, such that when the lines intercept, a targetis framed in another portion of the field of view of the camera 100 forexample.

The subject 405 of FIG. 4 is therefore intended to be substantiallycentrally framed within the field of view of the camera, generallydepicted by the dashed box 407. However, with reference to FIG. 4, it isclear that when the image was captured, deviation of the camera hascaused the subject 405 to be framed elsewhere, and therefore less thanoptimally according to the above characterisation.

In order to determine, or at least provide a probability of the framingwhich was desired, the state data and/or captured image data is analysedby processor 105. In a preferred embodiment, a memory of the camera 100(not shown) is operable to store data representing a number of usemodels of the camera. The models may be hidden Markov models forexample, or other suitable models, such as probabilistic models forexample. State data generated using the sensor 103 is processed inassociation with the models in order to determine a particular use modelof the camera 100 which has resulted in the generation of image datarepresenting a captured image, such as 400 of FIG. 4.

In the case that such use models are represented using hidden Markovmodels (HMMs), suitable HMMs may be generated in known ways usingexemplary data to train the HMMs by marking up image capture behavioursby test subjects, for example. Alternative ways of representing a usemodel of the camera 100 are possible as will be appreciated by thoseskilled in the art. For example, various use models can be representedby data in a look-up table or similar (either in hardware or software).

State data generated by the sensor 103 during the preview period ascharacterised by FIG. 2 and the related description above is used inorder to determine a probability that a particular use model hasresulted in the generation of image data representing the image 400 ofFIG. 4.

Once the probability of a use model has been generated, processor 105can process image data generated using the image capture element 101 asappropriate in order to compensate for any undesirable framing of theimage as a result of the use model which has been determined to be theone with the highest probability.

With reference to FIGS. 1 to 4, the target 405 of an image 400 isdetermined by processing data from the sensor 103,301 of the camera100,300 in association with captured image data 305 as necessary. Asdescribed, the data 307 represents a condition of the camera 100,300during the preview period of an image capture operation as characterisedby the periods 201,203 of FIG. 2. The subject of the image 400 isdetermined to be 405. Further, as explained, the state data 307 is alsoused, in association with a stored model, to determine a particular usemodel of the camera 100,300 which has resulted in the capture of theimage 400.

So, for example, in FIG. 4, and with reference to FIG. 2, the subject405 of the image 400 is stationary with respect to time in the dimensionbeing considered in FIG. 2, resulting in 209. State data 307 relating tothe periods 201,203 of FIG. 2 is processed by processor 103,309 inassociation with the captured image data 305 representing image 400, anda probability of the likely subject of the image 400 is determined,which is 405 in this case.

Preferably, the state data 307 processed by processor 105,309 providesan indication of the use model which has the highest probability ofbeing the appropriate use model based on the available state data 307and optionally with reference to the captured image data 305. The usemodel with the highest probability is selected by processor 105,309 asthat which has resulted in the generation of the image data. Followingdetermination of the use model, the processor 105,309 is operable todetermine a measure of the framing type and accuracy thereof, anddetermine based on this, whether any portions of the captured imageshould be automatically cropped in order to more clearly frame thesubject which was determined as being the desired subject of thecaptured image. So, for example, in FIG. 4, processing can be applied tothe captured image data in order to more clearly frame subject 405. Thiscan be done by cropping regions of the image 400 around subject 405 toprovide an output image with subject 405 substantially centrally framedtherein.

Stored data representing use models may include that representing a‘quick snap’, whereby a user has not taken care to frame the imageproperly perhaps due to time constraints or through lack of training, ormay include the scenario where a user has taken significant time inorder to frame a subject in a desired way. It will be appreciated thatmany other use models of a camera exist, and a suitable learned modelmay be stored in a memory of the camera 100,300 for each, with theprovision of adding to and/or removing such models a function of thecamera 100,300 according to a preferred embodiment.

As described above, if the processor 105,309 determines a use modelwhich corresponds to a user having effected care in capturing an image,an automatic image processing step of the camera 100,300 such asautocropping will not be applied to the relevant image. If however, thedetermined use model suggests that the user has not taken care and/ortime for example, then the processor 105,309 is operable to apply anautomatic image processing step to the image data 305 such asautocropping thereby generating output image data 311 which betterrepresents that which was intended to be captured by the user, i.e. thatin which the subject 405 is framed in a more desirable location of theimage 400, and data representing non-salient areas of the image isremoved.

Sensor 103,303 can generate data representative of a number of cameraproperties or changes therein. In addition to changes in camera positionand/or orientation, sensor 103,303 can additionally, or exclusively,generate data represented any one or more of the following: operation ofa zoom function of the camera including data relating to a length oftime with which the zoom function is operated; directional audio whichindicates, perhaps in combination with other features, that the user wasframing an animate subject and, in a preferred embodiment, if theframing action matches the direction of the sound it is likely to behighly relevant and well framed; motion in the scene being capturedwhich indicates, perhaps in combination with other features, that theuser was framing a moving object or perhaps was waiting for some motionto happen, generally indicated by very steady waiting and a finalshooting or motion when an object enters the scene; biometric clues,perhaps in combination with other features, such as heart rate, skinconductivity, pupil dilation, increased rate of breathing, changes inexpression etc. Other suitable indications are possible.

All the above points can also be used in order to determine a type offraming, in particular whether the object is centrally framed, or framedin a left or right handed portion of the field of view of the camera.For example, many users might move back and forth from the background tothe subject of the image in order to achieve a desired level ofexposure. Alternatively, when the user deliberately displaces a subjectin a certain portion of the camera field of view, the subject willtypically be framed before the user then moves the field of viewslightly. This situation typically occurs on cameras which require a“half-depress” of the shutter button to cause auto-exposure andauto-focus on the subject, before moving the camera to re-frame thefinal image. On such cameras, use of half-depress followed by re-framingis a natural sign of careful composition which can be detected by sensor103 and used to prevent an autocrop algorithm for example being appliedto generated image data.

Alternatively, the camera 100,300 can be arranged to capture an imagewith a slightly larger field of view than that shown to the user in theviewfinder of the camera or on any other suitable viewer of the camera.In the case of careless or less than optimal composition, an autocropalgorithm can generate a slightly larger field of view than the userselected in the situation where it appears that the edge of a salientregion of an image has been unintentionally clipped, or too tightlycropped in the user's framing. Also, in cases where the user hascomposed the image carefully, the excess pixels as a result of providingthe larger field of view can be discarded as required.

In a preferred embodiment, camera 100,300 can provide an indication to auser of the level of confidence of whether an image should beautomatically cropped, and by how much it should be cropped. Forexample, a measure of the level of confidence could be presented to auser a display of the camera 100,300. Such a measure could be displayedas continuous measure, for example a number in a specific range such asfrom 0 to 10, 0 indicating the lowest level of confidence, 10 indicatingthe highest level of confidence for example, or alternatively, themeasure could be expressed qualitatively using, for example, terms suchas “high”, “low” etc. Other alternatives are possible.

The present system and method provides the advantage that a camera willnot have to estimate quality, type or amount of framing of a subject ofan image, but will have objective clues coming from user behaviour whichwill enable the camera to determine such parameters. Hence, an autocropfunction of the camera can be included as a standard part of the imageprocessing, thereby simplifying the user interface and ensuring thatimages are automatically cropped when appropriate. When the user hascarefully framed an image by hand, this is detected automatically,thereby preventing the autocrop algorithm from changing the userscarefully considered image. The camera can include an override functionwhereby any automatic cropping of an image can be undone or prevented,or applied if desired.

A use model may be determined following image capture as describedabove, or alternatively, ‘on the fly’ such that an instantaneous measureof the type of use model of the camera 100,300 is substantiallycontinuously generated by processor 105,309 based on available generatedstate data 307, and optionally using image data registered using theimage capture element 101,301 of the camera 100,300. Preferably, statedata 307 is generated by sensor 103,303 continuously when the camera100,300 is powered and in a suitable mode of operation for image orvideo capture. A specific amount of memory may be allocated for storageof the state data such that old data is overwritten by new for example.The amount of storage can be determined based on considerationsincluding the total amount of memory available in the camera inquestion, the amount of state data required in order to provide areliable determination of a use model, and the accuracy of thedetermination required. Other considerations can be taken into account.

The generation of an instantaneous measure as described above can beeffected using state data generated up to the point where the measure isgenerated.

Such an instantaneous measure can be used in a number of ways. Forexample, the determined type of framing and accuracy thereof can bepresented to a user using a display (not shown) of the camera 100,300.The instantaneous measure can also be used to control image processingbefore, during and after capture as necessary, so that camera functionssuch as exposure control and deblurring for example, can be controlledon the basis of the instantaneous measure and/or the type of framingdetermined after image capture.

In addition to the above, a suitable memory (not shown) of the camera100,300 can store data representing a specific user of the camera, and atypical use model favoured by that user for example. Such data can bedynamically updated based on the use of the camera 100,300 by the userin question. Furthermore, the memory can store data representingdifferent framing contexts. Such data can be used to determine thedeviation from a standard or stored framing behaviour, e.g. an outdoorcycling situation as opposed to portrait situation.

The term “camera” is used above in order to describe implementations ofpreferred embodiments. It will be appreciated by those skilled in theart however, that this term is not intended to be limiting, and mayencompass not only conventional analogue or digital cameras orvideo-cameras, but also mobile devices such as mobile telephones, PDAs(personal digital assistants) and pagers and the like which have imagingfunctionality, and generally any mobile image capture device.

The term “framing” and related terms as used above relate to the processof moving/orienting a camera in order that a target, of which an imageis desired, is substantially within a desired portion of the field ofview of the camera, such as centrally within the field of view of thecamera for example.

The approach taken in provides an automated method of generatingconstraints or hints. By inferring appropriate constraints automaticallyfrom the behaviour of the camera user prior to image capture, a userdoes not have to explicitly interact with an autocrop (or other imageprocessing) algorithm of the camera. In cases where a user has obviouslytaken considerable care to compose an image carefully, an autocropalgorithm can be completely disabled.

The advantages are that the system will not have to guess on thequality, type or amount of framing, but will have objective clues comingfrom user behaviour to determine such parameters. An autocrop facilitycan be safely included as a standard part of the image processing,thereby simplifying the user interface and ensuring that images arecropped when appropriate. When the user has carefully framed an image byhand, this is detected automatically, thereby preventing the autocropalgorithm from changing the users carefully considered image.

The patterns of both motion, time and camera control during a previewstage can be monitored in order to determine the accuracy of theframing, which can be expressed qualitatively(none/low/medium/high/perfect) or with a continuous measurer (e.g.0-255). The analysis can be done using a learned (probabilistic) model,e.g. Hidden Markov Models, which can be trained by marking up capturebehaviours by test subjects. The analysis could be performed by usingindividual behaviour features extracted independently from rawmeasurements, sensor or camera controls. One feature could be time toshoot, which indicates, perhaps in combination with other features, ahasty framing as opposed to a careful one. Another feature could beoperation of the zoom, which indicates, perhaps in combination withother features, that the user was interested in a particular subject andhence the framing would tend to have been done with high care as well asbeing precise in this case. Another feature could be the motion of thecamera, which pattern indicates, perhaps in combination with otherfeatures, whether the user went through the “servoed” phase (see above)or not, indicating that the framing was done with care. Another featurecould be length of time over which the zoom was operated, whichindicates, perhaps in combination with other features, especiallymotion, that the user was interested in a particular subject and hencethe framing would tend to have been done with high care as well asprecision in this case. Another feature could be directional audio,which indicates, perhaps in combination with other features, that theuser was framing an animate subject and if the framing action generallymatches the direction of the sound it is likely to be highly relevantand well framed. Another feature could be motion in the scene, whichindicates, perhaps in combination with other features, that the use wasframing a moving object or perhaps was waiting for some motion tohappen, indicated by very steady waiting and a final shooting or motionwhen an object enters the scene. Another feature could be biometricclues, perhaps in combination with other features; for instance it isunlikely that a startled user will have framed an image well.

All the above points can be used to determine also the type of framing,in particular whether the object is central or left and right forexample; for instance many users might hop back and forth from thebackground to the subject in order to find the right exposure etc.Alternatively, when the user deliberately displaces a subject, say tothe left, they will typically frame the subject, and then “jump away”slightly and readjust. This situation typically occurs on cameras whichrequire a “half-depress” of the shutter button (image capture actuator)to cause auto-exposure and auto-focus on the subject, before moving thecamera to re-frame the final image. On such cameras, use of half-depressfollowed by re-framing is a natural sign of careful composition whichcan be detected and used to prevent autocrop (or other imageprocessing).

A camera could be arranged to capture an image with a slightly widerfield of view than that shown to the user in the viewfinder. In the caseof careless composition, an autocrop algorithm can therefore have theability to generate a slightly larger field of view than the userselected, in the situation where it appears that the edge of a salientregion has been unintentionally clipped, or too tightly cropped in theuser's framing. In cases where the user has composed the imagecarefully, the excess pixels can be discarded.

Using all the above techniques, it is possible to give an indication ofconfidence of whether, and by how much, to automatically crop a picture.For very accurate framing auto-cropping is likely to be wrong as theuser has taken care in framing.

The above techniques can be used to control any image processingperformed after capture, e.g. exposure control deblurring etc.Alternatively, the above techniques can be used to control internalcamera settings before exposure.

A status of the framing accuracy and/or type can be indicated to a userof a camera using a suitable display of the camera for example/

Also, a camera can be adapted to recognize a “framing fingerprint” of aparticular user, i.e. the way in which a user typically uses the camera.The camera can then automatically process image data on the basis of thefingerprint. A camera can also be adapted to recognize the “framingcontext” by analyzing any substantial deviation from standard or storedframing behaviours, e.g. an outdoor cycling situation as opposed to aportrait situation where framing is done better.

The invention claimed is:
 1. A method of processing image datarepresenting a captured image, the data generated using an image captureelement of an image capture device, the method comprising: generatingstate data representing at least one characteristic of motion of theimage capture device and a status of at least one setting of the deviceduring a predetermined time period, wherein the predetermined timeperiod is a time period before the image is captured; and processing theimage data on the basis of the state data in order to provide a measureof the accuracy of framing of the captured image within a field of viewof the device.
 2. A method as claimed in claim 1, further comprising:processing the image data on the basis of the state data in order toprovide a measure of the type of framing of the captured image within afield of view of the device.
 3. A method as claimed in claim 1, whereinthe measure is a qualitative measure.
 4. A method as claimed in claim 1,wherein the measure is a quantitative measure.
 5. A method as claimed inclaim 1, wherein processing the image data includes comparing the statedata with data representing at least one model, the or each modelrepresenting at least one exemplary motion pattern and/or device settingpattern of the device, and processing the image data on the basis of thecomparison.
 6. A method as claimed in claim 5, wherein the model data isat least one hidden Markov model.
 7. A method as claimed in claim 5,wherein comparing the state data further includes determining a modelwith the highest probability of representing the characteristic ofmotion of the image capture device and status of the or each setting ofthe device during the predetermined period.
 8. A method as claimed inclaim 1, further comprising generating temporal data representing a timeframe over which the image was captured, and processing generated imagedata on the basis of the temporal data.
 9. A method as claimed in claim1, wherein determining the status of at least one setting of the deviceincludes generating zoom data representing a status of a zoom functionof the device during the predetermined time period, and processing theimage data on the basis of the zoom data.
 10. A method as claimed inclaim 1, further comprising: detecting an audio signal using an audiosensor of the device; generating directional audio data representing adirection with respect to the audio sensor from which the audio signalwas detected; and processing generated image data on the basis of thedirectional audio data.
 11. A method as claimed in claim 1, furthercomprising: determining a subject of an image during the predeterminedperiod; generating image subject data representing a motion of a subjectof the image within a field of view of the device during the period; andprocessing the image data on the basis of the image subject data.
 12. Amethod as claimed in claim 1, further comprising: generating biometricdata using at least one biometric sensor of the device, the biometricdata representing any one or more of skin conductivity, heart rate,pupil dilation, blood pressure, or perspiration of a user of the device;and processing the image data on the basis of the biometric data.
 13. Amethod as claimed in claim 1, further comprising: displaying a measureof the framing type and/or accuracy to a user of the device.
 14. Amethod as claimed in claim 13, wherein the framing data representswhether the subject of the image is framed substantially centrally, orin a left hand, right hand, and/or upper or lower portion of the fieldof view.
 15. A method as claimed in claim 1, wherein determining thestatus of at least one device setting further comprises: generatingauto-focus data representing whether an image capture actuator of thedevice, the actuator operable to effect image capture, is in a statecausing auto-focus of the device on a desired subject, followed bymovement of the device in order to re-frame the desired subject whilstin focus, in the predetermined period; and processing generated imagedata on the basis of the auto-focus data.
 16. A method as claimed inclaim 1, wherein processing the generated image data includes removing aportion of the image.
 17. A method as claimed in claim 16, whereinremoving a portion of the image comprises: determining a subset of imagedata representing a portion of the capture image, the portion includingat least a region of the image comprising a desired subject of theimage; and deleting image data outside of the subset from a memory ofthe device to provide output image data representing a desired image.18. A method as claimed in claim 1, wherein the image capture element isadapted to generate image data from a larger field of view of the devicethan that viewable by a user of the device, and wherein processing theimage data representing a captured image includes using image data froma portion of the field of view not viewable by a user of the device. 19.A method as claimed in claim 1, further comprising: alerting the user ofthe device that the measure of the accuracy of the framing is below apredetermined threshold value.
 20. A method as claimed in claim 1,wherein the device is adapted to accept input from a user, the inputoperable to override processing of the image data.
 21. A method asclaimed in claim 1, further comprising controlling at least one camerafunction on the basis of the state data.
 22. A method as claimed inclaim 8, wherein the temporal data forms part of the state data.
 23. Animage capture device comprising: an image capture element operable togenerate image data; a processor; and a state data generator forgenerating state data representing at least one characteristic of motionof the image capture device and a status of at least one setting of thedevice during a predetermined time period, wherein the processor isoperable to process image data representing a captured image on thebasis of the state data in order to provide a measure of the accuracy offraming of the captured image within a field of view of the device, andwherein the predetermined time period is a time period before the imageis captured.
 24. A device as claimed in claim 23, further comprising amemory comprising data representing at least one probabilistic model,the or each model representing at least one exemplary motion patternand/or device setting pattern of the device.
 25. A device as claimed inclaim 23, further comprising an image capture actuator operable toeffect capture of an image, the processor operable to generateauto-focus data representing whether the actuator is in a state causingauto-focus of the device on a desired subject, followed by movement ofthe device in order to re-frame the desired subject whilst in focus, inthe predetermined period, and process generated image data on the basisof the auto-focus data.
 26. A device as claimed in claim 23, wherein thedevice is operable to automatically focus on a subject within a field ofview of the device.
 27. A device as claimed in claim 23, wherein a userof the device effects focus of the device on a desired subject within afield of view of the device using a focus actuator of the device.
 28. Adevice as claimed in claim 23, wherein the processor is operable togenerate data representing a measure of a type of framing of a capturedimage within a field of view of the device.
 29. An image capture deviceoperable in accordance with the method as claimed in claim
 1. 30. Anon-transitory computer readable medium having stored thereon a computerexecutable program, the computer executable program when executed causesa computer system to: generate state data representing at least onecharacteristic of motion of an image capture device and a status of atleast one setting of the device during a predetermined time period; andprocess image data on the basis of the state data in order to provide ameasure of the type and/or accuracy of framing of a captured imagewithin a field of view of the device, wherein the predetermined timeperiod is a time period before the image is captured.
 31. Anon-transitory computer readable medium having stored thereon a computerexecutable program, the computer executable program when executed causesa computer system to process image data in accordance with the methodclaimed in claim
 1. 32. A hardware logic circuit configured to operatein accordance with the method claimed in claim
 1. 33. A method asclaimed in claim 9, wherein the zoom data forms part of the state data.34. A method as claimed in claim 10, wherein the audio data forms partof the state data.
 35. A method as claimed in claim 11, wherein thesubject data forms part of the state data.
 36. A method as claimed inclaim 12, wherein the biometric data forms part of the state data.
 37. Amethod as claimed in claim 14, wherein the framing data forms part ofthe state data.
 38. A method as claimed in claim 15, wherein theauto-focus data forms part of the state data.